Skip to content

fix(ci): verify committed codegen output instead of a regenerated worktree - #177

Merged
auto-wood merged 1 commit into
mainfrom
fix/ci-verify-committed-codegen
Nov 16, 2026
Merged

auto-wood merged 1 commit into
mainfrom
fix/ci-verify-committed-codegen

Conversation

@auto-wood

Copy link
Copy Markdown
Contributor

Follow-up to #176 review threads.

Thread 2 (should-fix) — fixed

verify-codegen ran codegen first, so the typecheck and cargo check that followed compiled freshly rewritten worktree files, not the committed trees that publish as @42ch/spoke-schemas / spoke-schemas. A schema edit without a committed regeneration passed.

It now verifies the committed trees: schema-count and opaque-JSON asserts, the deep-ref-hoist test, @42ch/spoke-schemas typecheck, then cargo check --workspace. Regeneration is the author step (pnpm run codegen); no byte diff returns.

Threads 1 and 3 (should-fix) — resolved by documenting the chosen coverage

Removing the committed C carrier hash gate was the stated goal of #176: CI must not fail on compiled-byte identity. Those threads are addressed by making the docs state the actual contract instead of demanding a check that no longer exists:

  • provenance.json hashes are build audit data, not a CI gate.
  • Committed carriers are covered by the C ABI symbol check and the C++ smoke (behavior).
  • CONTRIBUTING.md no longer says the recorded hashes must match committed bytes.

Threads 4, 5, 6 (may-fix) — fixed

  • Removed the orphaned two-tree comparison instruction (already gone from main; this PR keeps refresh text on the transfer manifest).
  • Codegen knowledge and index no longer claim a zero-drift proof covers generated bytes.
  • Reproducibility is scoped to identical inputs (source, lockfile, toolchain, target, output path), matching the tooling-decision record.

Verification

  • node tooling/codegen/assert-schema-count.mjs — OK
  • node tooling/codegen/assert-opaque-json-types.mjs — OK
  • node --test tooling/codegen/test/deep-ref-hoist.test.mjs — pass
  • pnpm -F @42ch/spoke-schemas typecheck — pass
  • cargo +nightly check --workspace — pass
  • node tooling/docs/twin-parity.mjs — OK

@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

verify-codegen now checks committed generated sources without regenerating them first.

  • Codegen checks now run against the committed TypeScript and Rust output.
  • Native artifact guidance separates build hashes from behavior checks.

No actionable issues found. auto-wood explicitly states that compiled-byte identity is not a CI requirement and that recorded hashes remain audit data.

Reviews (1) · Last reviewed commit: "fix(ci): verify committed codegen output..." · Reviewed by Greptile

@m-inspector

m-inspector Bot commented Nov 16, 2026

Copy link
Copy Markdown

✅ Verdict: ship it · Score 88% (good)

Findings Count
🔴 must-fix 0
🟠 should-fix 0
🟡 may-fix 2
🔵 nit 1

📋 Review

What this PR does: This PR is a follow-up to #176 that reshapes verify-codegen: it drops the leading pnpm run codegen step so the chain validates the committed generated trees (schema count, opaque-JSON contract, deep-ref-hoist test, @42ch/spoke-schemas typecheck, cargo check --workspace) instead of compiling freshly regenerated worktree files. Alongside it the PR rewrites the surrounding documentation so the stated contract matches what is actually gated — codegen as an author step, provenance.json hashes as build audit data rather than a CI gate, committed C carriers covered by the symbol check and C++ smoke, and reproducibility scoped to identical inputs. The settled verdict is ship it at 88% (good) with no must-fix or should-fix items; what remains is one knowledge-doc posture correction, one residual wording nit, and the question of whether removing the generator from CI should be paired with a re-added generatability or dirty-tree check.

Findings

🟡 Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/

  • Evidence:

    [DOCS-01] Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/

    • Evidence: .agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31 — Decision step 3 now reads: "pnpm run verify-codegen checks the committed trees: tooling/codegen/assert-schema-count.mjs → tooling/codegen/assert-opaque-json-types.mjs → node --test …deep-ref-hoist.test.mjs → @42ch/spoke-schemas typecheck → cargo check --workspace. Regeneration is the author step (pnpm run codegen); CI verifies the committed output that ships, never a regenerated worktree."
    • Evidence: tooling/codegen/assert-schema-count.mjs:29-31 — EXPECTED_SCHEMA_COUNT = 114; the script only counts schemas/**/*.schema.json files and never enumerates packages/spoke-schemas/src/generated or the Rust generated tree, so a schema body edit that skips regeneration leaves the count at 114 and passes.
    • Evidence: tooling/codegen/assert-opaque-json-types.mjs:1-20 — the checker reads committed generated TS (packages/spoke-schemas/src/generated) and asserts the opaque-JSON shape; it is self-consistent against stale output, not proof of freshness.
    • Evidence: .github/workflows/ci.yml:87-89 — the verify-codegen job runs only pnpm run verify-codegen; .github/workflows/release.yml:163-164 does the same before publish. Neither adds a post-step git diff --exit-code, and grep across .github/workflows/ finds no other codegen invocation.
    • Impact: The doc is the durable tooling fact surface loaded before touching schemas/. Its "CI verifies the committed output that ships" wording, placed immediately after removal of the regeneration step, tells a reader that a green verify-codegen certifies the committed trees are current with the hand-authored SSOT declared in the same doc's Problem statement ("must produce both TypeScript and Rust without drift"). No step in the chain does that: typecheck and cargo check comp…
  • Location: .agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31

  • Category: DOCS

  • Merge class: may-fix

🔵 README index summary drops the generated-tree freshness dimension the row previously carried

  • Evidence:

    [DOCS-02] README index summary drops the generated-tree freshness dimension the row previously carried

    • Evidence: .agents/knowledge/README.md:9 — the codegen row summary changed from "verify-codegen regenerates both trees, asserts the schema count and opaque-JSON types, then runs cargo check --workspace" to "verify-codegen checks the committed trees — schema count, opaque-JSON types, TypeScript typecheck and cargo check --workspace".
    • Impact: Both halves of the new wording are factually correct against the current script (package.json:15: assert-schema-count → assert-opaque-json-types → deep-ref-hoist test → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace) and against packages/spoke-schemas/package.json:37 (typecheck: tsc --noEmit exists). The residual reader cost is only that the index row now reads as a complete verification list while the freshness question (see DOCS-01) is the same one; this is a wording consequence of the same change, not an independent wrong outcome.
    • Effort: XS
    • Risk: LOW — summary text only.
    • Confidence: HIGH (opened the row and both scripts).
    • Merge class: nit
    • Fix sketch: Align the summary row's wording with the corrected posture once DOCS-01 is settled; no behavior change.
  • Location: .agents/knowledge/README.md:9

  • Category: DOCS

  • Merge class: nit

🟡 verify-codegen no longer executes the generator, so CI never proves the schema set is still generatable

  • Evidence:
    • Evidence: package.json:15 — the changed verify-codegen chain is now assert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace; the leading pnpm run codegen was deleted. tooling/codegen/assert-schema-count.mjs:6-9 still describes the author procedure as "Run pnpm run codegen; commit schema + generated output + this constant in one commit. pnpm run verify-codegen must PASS", so step 4 no longer executes step 3. The two CI consumers run this exact chain: .github/workflows/ci.yml:87-89 ("Verify codegen output": pnpm run verify-codegen) and .github/workflows/release.yml:163-164. tooling/codegen/assert-opaque-json-types.mjs:14 ("Wire it from pnpm run verify-codegen") is unaffected because that script reads schemas/, not generated output.
    • Impact: Before this change the CI job proved two things — the committed trees compile and schemas/ still generates. After it, the generator (tooling/codegen/src/run.mjs → jstt + typify, plus rust-gen's exact-114 fail-fast) is exercised by no CI job at all. Concretely: a PR that edits schemas/**/*.schema.json, fails to regenerate (the case the new docs warn about), and commits stale-but-compiling generated output now passes verify-codegen green, whereas the previous chain failed at codegen/rust-gen for any schema the generators reject. The committed generated trees that ship (and that docs/reference/protocol.md:23 / docs/zh/reference/protocol.md:23 now describe as "checked") can therefore disagree with the hand-authored SSOT with no red check. Note the pre-existing part: drift itself was never asserted (no git diff --exit-code after regeneration), so this finding is limited to the newly removed generatability guard, not to a new drift detector that never existed.
    • Effort: XS — restore one pnpm run codegen step, or run it in a throwaway state and assert a clean tree.
    • **R…
  • Location: package.json:15
  • Category: dx
  • Merge class: may-fix
Revalidated findings

none

Linked-issue AC

none — the PR links no issue (the issues array is empty); only #176 review threads are referenced in prose, which carry no testable acceptance criteria here.

✅ Verified

- Read package.json:15: the changed verify-codegen chain is assert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace, with the leading pnpm run codegen removed.
- Read .github/workflows/ci.yml:87-89 and .github/workflows/release.yml:163-164: both invoke only pnpm run verify-codegen, with no post-step git diff --exit-code; a grep across .github/workflows/ found no other codegen invocation.
- Read tooling/codegen/assert-schema-count.mjs in full: EXPECTED_SCHEMA_COUNT = 114, counting only schemas/**/*.schema.json and never enumerating packages/spoke-schemas/src/generated or the Rust generated tree.
- Read tooling/codegen/assert-opaque-json-types.mjs:1-20: the checker reads committed generated TS, so it is self-consistent against stale output rather than proof of freshness.
- Read packages/spoke-schemas/package.json:37: a typecheck: tsc --noEmit script exists, so the typecheck leg of the new chain is real.
- Read .agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31 and .agents/knowledge/README.md:9 in full, checking each rewritten sentence against the scripts above.
- ❓ The PR-body verification commands (assert-schema-count, assert-opaque-json-types, deep-ref-hoist test, spoke-schemas typecheck, cargo +nightly check --workspace, twin-parity.mjs) are author-claimed only; this seat read statically and did not re-execute them.

🗑️ Considered & rejected

- A "new drift detector" regression: rejected as a new defect — no git diff --exit-code against generated paths ever existed in this chain, so the PR changes generatability coverage, not drift detection; the finding is scoped to the removed codegen step only.
- The README index row (.agents/knowledge/README.md:9) as a substantive wrongness: rejected — both halves of the new wording check out against package.json:15 and packages/spoke-schemas/package.json:37; it survives only as a nit that trails DOCS-01's wording outcome.
- Publishing stale-but-compiling trees from release.yml as an independent finding: rejected — it is the same root cause as the removed generator step and is folded into that one finding rather than counted twice.
- provenance.json hash / committed-carrier coverage concerns (PR threads 1 and 3): rejected — removal of the C carrier hash gate was the stated goal of #176 and the PR resolves those threads by documenting the actual contract; nothing in this seat's file scope contradicts that posture.
- The stale rustfmt rationale at .github/workflows/ci.yml:73 (comment and component install now unjustified): noted but not elevated — it is cross-domain residue explicitly outside this seat's file scope, recorded in the fix sketch for the workflow reviewer.
- Any must-fix or should-fix escalation: rejected — the two may-fix items are documentation and posture accuracy, touching no runtime behavior, and the verdict stands at ship it.

🛠️ Plan to fix

Expand fix plan
1. **DOCS-01 — correct the verification posture in the pipeline doc** (`.agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31`, plus the nearby step-4 `rust-gen` "exactly 114 Rust schema modules" clause). Replace "CI verifies the committed output that ships" with wording that states precisely what the chain proves — count, opaque-JSON contract, TS typecheck, Rust compile, all properties of the committed bytes — and that freshness against `schemas/` is author discipline (`pnpm run codegen` + commit), not a CI gate. Make the Verify list and step 4 consistent with that posture so a reader no longer infers a zero-drift proof.
2. **Decide the generatability posture** (`package.json:15`, `.github/workflows/ci.yml:87-89`, `.github/workflows/release.yml:163-164`). Either (a) restore `pnpm run codegen && git diff --exit-code -- packages/spoke-schemas/src/generated crates/spoke-schemas/src/generated`, keeping both workflow consumers on the same chain and accepting the `cargo run --release` cost of `rust-gen`, or (b) keep the current author-only posture and record it explicitly in the codegen knowledge/pipeline doc: no CI job exercises `tooling/codegen/src/run.mjs`, jstt, typify or `rust-gen`'s fail-fast, so a reviewer must run `pnpm run codegen` by hand. Whichever is chosen, update the author-procedure comment at `tooling/codegen/assert-schema-count.mjs:6-9` so step 4 no longer claims to execute step 3.
3. **DOCS-02 — align the index summary last** (`.agents/knowledge/README.md:9`): once DOCS-01's final wording is settled, make the codegen row's summary reflect the same posture (properties verified vs. author-only freshness). No behavior change.
4. **Workflow residue handoff**: flag to the workflow owner that `.github/workflows/ci.yml:73` still installs the `rustfmt` component for a comment claiming `verify-codegen` runs `rustfmt` on generated memory types, which `cargo check` no longer does — outside this seat's scope, but it should land in the same change or be tracked.
5. **Verification gates after the edits**: `node tooling/codegen/assert-schema-count.mjs`, `node tooling/codegen/assert-opaque-json-types.mjs`, `node --test tooling/codegen/test/deep-ref-hoist.test.mjs`, `pnpm -F @42ch/spoke-schemas typecheck`, `cargo +nightly check --workspace`, plus `node tooling/docs/twin-parity.mjs`; if posture (a) is chosen, additionally confirm both workflows run the extended chain and that a deliberately unregenerated schema edit turns red. Re-run the doc twin-parity check after reconciling the `docs/reference/protocol.md:23` / `docs/zh/reference/protocol.md:23` "checked" wording with the chosen posture.

Review 1 · commit 9068d63

@m-inspector m-inspector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mstar-inspector line comments · round 1 · 9068d63

1. **SSOT** — only `schemas/**/*.schema.json` are hand-authored; generated trees are committed.
2. **Orchestrator** — `tooling/codegen` (`pnpm -F @42ch/spoke-codegen codegen` → `node tooling/codegen/src/run.mjs`) walks schemas, localizes `$ref`s for typify, and emits mirrored `generated/{common,connect,data,ops}/` trees in both packages.
3. **Verify** — `pnpm run verify-codegen` runs `codegen` → `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `cargo check --workspace`.
3. **Verify** — `pnpm run verify-codegen` checks the committed trees: `tooling/codegen/assert-schema-count.mjs` → `tooling/codegen/assert-opaque-json-types.mjs` → `node --test tooling/codegen/test/deep-ref-hoist.test.mjs` → `@42ch/spoke-schemas` typecheck → `cargo check --workspace`. Regeneration is the author step (`pnpm run codegen`); CI verifies the committed output that ships, never a regenerated worktree.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/ · 🟡 may-fix

[DOCS-01] Knowledge doc presents CI as verifying committed generated trees, but no chain step compares them to schemas/

  • Evidence: .agents/knowledge/architecture-patterns/spoke-codegen-pipeline.md:31 — Decision step 3 now reads: "pnpm run verify-codegen checks the committed trees: tooling/codegen/assert-schema-count.mjs → tooling/codegen/assert-opaque-json-types.mjs → node --test …deep-ref-hoist.test.mjs → @42ch/spoke-schemas typecheck → cargo check --workspace. Regeneration is the author step (pnpm run codegen); CI verifies the committed output that ships, never a regenerated worktree."
  • Evidence: tooling/codegen/assert-schema-count.mjs:29-31 — EXPECTED_SCHEMA_COUNT = 114; the script only counts schemas/**/*.schema.json files and never enumerates packages/spoke-schemas/src/generated or the Rust generated tree, so a schema body edit that skips regeneration leaves the count at 114 and passes.
  • Evidence: tooling/codegen/assert-opaque-json-types.mjs:1-20 — the checker reads committed generated TS (packages/spoke-schemas/src/generated) and asserts the opaque-JSON shape; it is self-consistent against stale output, not proof of freshness.
  • Evidence: .github/workflows/ci.yml:87-89 — the verify-codegen job runs only pnpm run verify-codegen; .github/workflows/release.yml:163-164 does the same before publish. Neither adds a post-step git diff --exit-code, and grep across .github/workflows/ finds no other codegen invocation.
  • Impact: The doc is the durable tooling fact surface loaded before touching schemas/. Its "CI verifies the committed output that ships" wording, placed immediately after removal of the regeneration step, tells a reader that a green verify-codegen certifies the committed trees are current with the hand-authored SSOT declared in the same doc's Problem s…

| Doc | Category | Source | Summary |
|-----|----------|--------|---------|
| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` regenerates both trees, asserts the schema count and opaque-JSON types, then runs `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — compiling generated bytes does not cover downstream handlers (land ABI + consumer updates together) |
| [`architecture-patterns/spoke-codegen-pipeline.md`](architecture-patterns/spoke-codegen-pipeline.md) | architecture-patterns | current `schemas/**/*.schema.json`, `tooling/codegen` and generated trees | jstt+typify orchestrator over **114** hand-authored schemas; each generated tree is **114 modules + 5 barrels = 119 files**; `verify-codegen` checks the committed trees — schema count, opaque-JSON types, TypeScript typecheck and `cargo check --workspace`; rust-gen fail-fast; **presence-sensitive arrays** — six byte-bound modules module-wide, plus `Content.cognition` in all 15 carrier modules / 17 clone types (unrelated arrays keep `Vec` + `is_empty`); typify nominal-clone canonical-import guide and Builder construction; **regeneration ripple** — compiling generated bytes does not cover downstream handlers (land ABI + consumer updates together) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README index summary drops the generated-tree freshness dimension the row previously carried · 🔵 nit

[DOCS-02] README index summary drops the generated-tree freshness dimension the row previously carried

  • Evidence: .agents/knowledge/README.md:9 — the codegen row summary changed from "verify-codegen regenerates both trees, asserts the schema count and opaque-JSON types, then runs cargo check --workspace" to "verify-codegen checks the committed trees — schema count, opaque-JSON types, TypeScript typecheck and cargo check --workspace".
  • Impact: Both halves of the new wording are factually correct against the current script (package.json:15: assert-schema-count → assert-opaque-json-types → deep-ref-hoist test → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace) and against packages/spoke-schemas/package.json:37 (typecheck: tsc --noEmit exists). The residual reader cost is only that the index row now reads as a complete verification list while the freshness question (see DOCS-01) is the same one; this is a wording consequence of the same change, not an independent wrong outcome.
  • Effort: XS
  • Risk: LOW — summary text only.
  • Confidence: HIGH (opened the row and both scripts).
  • Merge class: nit
  • Fix sketch: Align the summary row's wording with the corrected posture once DOCS-01 is settled; no behavior change.

Comment thread package.json
"scripts": {
"codegen": "pnpm -F @42ch/spoke-codegen codegen",
"verify-codegen": "pnpm run codegen && node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && cargo check --workspace",
"verify-codegen": "node tooling/codegen/assert-schema-count.mjs && node tooling/codegen/assert-opaque-json-types.mjs && node --test tooling/codegen/test/deep-ref-hoist.test.mjs && pnpm -F @42ch/spoke-schemas typecheck && cargo check --workspace",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verify-codegen no longer executes the generator, so CI never proves the schema set is still generatable · 🟡 may-fix

  • Evidence: package.json:15 — the changed verify-codegen chain is now assert-schema-count.mjs → assert-opaque-json-types.mjs → node --test deep-ref-hoist.test.mjs → pnpm -F @42ch/spoke-schemas typecheck → cargo check --workspace; the leading pnpm run codegen was deleted. tooling/codegen/assert-schema-count.mjs:6-9 still describes the author procedure as "Run pnpm run codegen; commit schema + generated output + this constant in one commit. pnpm run verify-codegen must PASS", so step 4 no longer executes step 3. The two CI consumers run this exact chain: .github/workflows/ci.yml:87-89 ("Verify codegen output": pnpm run verify-codegen) and .github/workflows/release.yml:163-164. tooling/codegen/assert-opaque-json-types.mjs:14 ("Wire it from pnpm run verify-codegen") is unaffected because that script reads schemas/, not generated output.
  • Impact: Before this change the CI job proved two things — the committed trees compile and schemas/ still generates. After it, the generator (tooling/codegen/src/run.mjs → jstt + typify, plus rust-gen's exact-114 fail-fast) is exercised by no CI job at all. Concretely: a PR that edits schemas/**/*.schema.json, fails to regenerate (the case the new docs warn about), and commits stale-but-compiling generated output now passes verify-codegen green, whereas the previous chain failed at codegen/rust-gen for any schema the generators reject. The committed generated trees that ship (and that docs/reference/protocol.md:23 / docs/zh/reference/protocol.md:23 now describe as "checked") can therefore disagree with the hand-authored SSOT with no red check. Note the pre-existing part: drift itself was never asserted (no git diff --exit-code after regeneration), so this finding is limited to the newly removed generatability guard, not to a new drift detector that never exis…

@auto-wood
auto-wood merged commit 371b5c2 into main Nov 16, 2026
27 of 28 checks passed
@auto-wood
auto-wood deleted the fix/ci-verify-committed-codegen branch November 16, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants